ARROW-16219: [CI] Fix git config to prevent SCM tools failure #12945

raulcd · 2022-04-21T13:10:24Z

This PR fixes the CI failures due to the latest git release fixing CVE-2022-24765.
I have been able to see the build passing the scm step with the change here:
https://app.travis-ci.com/github/raulcd/arrow/builds/249688925
The above build fails due to some tests failing but not related with installation anymore.

And this was the failure before the change:
https://app.travis-ci.com/github/raulcd/arrow/builds/249683847

I also have been able to reproduce the issue on the verify-conda-rc locally.

The failure:

$ docker-compose run  conda-verify-rc

....
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
/arrow/python /arrow /
Traceback (most recent call last):
File "/arrow/python/setup.py", line 607, in <module>
  setup(
File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
  return distutils.core.setup(**attrs)
File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 109, in setup
  _setup_distribution = dist = klass(attrs)
File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools/dist.py", line 462, in __init__
  _Distribution.__init__(
File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 293, in __init__
  self.finalize_options()
File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools/dist.py", line 886, in finalize_options
  ep(self)
File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools/dist.py", line 907, in _finalize_setup_keywords
  ep.load()(self, ep.name, value)
File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools_scm/integration.py", line 75, in version_keyword
  _assign_version(dist, config)
File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools_scm/integration.py", line 51, in _assign_version
  _version_missing(config)
File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools_scm/__init__.py", line 106, in _version_missing
  raise LookupError(
LookupError: setuptools-scm was unable to detect version for /arrow.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj
Failed to verify release candidate. See /tmp/arrow-HEAD.YUVPq for details.

Waiting to validate the verify-rc fix at the moment, will update once the local build finishes

…g the version

raulcd · 2022-04-21T13:20:06Z

We can see on the new build how the ARROW_GIT_ID variable is correctly populated now too:

--   ARROW_GIT_ID=f94a14ddd3a5271a3779b831da482dfe212d57a7 [default=""]
--       The Arrow git commit id (if any)

…s git safe directory

raulcd · 2022-04-21T13:48:22Z

Reproduced the issue and fixed on verify-release-candidate too:

$ docker-compose run  conda-verify-rc
...
Executing transaction: done
/arrow/python /arrow /
running build_ext
-- Running cmake for pyarrow
cmake -DPYTHON_EXECUTABLE=/tmp/arrow-HEAD.fNS4t/mambaforge/envs/conda-source/bin/python -DPython3_EXECUTABLE=/tmp/arrow-HEAD.fNS4t/mambaforge/envs/conda-source/bin/python "" -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=on -DPYARROW_BUILD_GANDIVA=on -DPYARROW_BUILD_DATASET=on -DPYARROW_BUILD_ORC=on -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PARQUET_ENCRYPTION=on -DPYARROW_BUILD_PLASMA=on -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=on -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release /arrow/python
CMake Warning:
  Ignoring empty string ("") provided on the command line.


-- The C compiler identification is GNU 10.3.0
-- The CXX compiler identification is GNU 10.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /tmp/arrow-HEAD.fNS4t/mambaforge/envs/conda-source/bin/x86_64-conda-linux-gnu-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /tmp/arrow-HEAD.fNS4t/mambaforge/envs/conda-source/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- System processor: x86_64
-- Performing Test CXX_SUPPORTS_SSE4_2
-- Performing Test CXX_SUPPORTS_SSE4_2 - Success
-- Performing Test CXX_SUPPORTS_AVX2
-- Performing Test CXX_SUPPORTS_AVX2 - Success
-- Performing Test CXX_SUPPORTS_AVX512
-- Performing Test CXX_SUPPORTS_AVX512 - Success
-- Arrow build warning level: PRODUCTION
Using ld linker

raulcd · 2022-04-21T13:54:31Z

@jorisvandenbossche @pitrou I've been able to validate on travis for s390x Python and locally for conda-verify-rc the fix as shown on the comments. We probably want to wait until the build passes but should be ready for review.

dev/release/verify-release-candidate.sh

github-actions · 2022-04-21T15:25:37Z

https://issues.apache.org/jira/browse/ARROW-162219

…idate script and add it to the docker-compose file

raulcd · 2022-04-21T15:35:39Z

@github-actions crossbow submit conda-verify-rc

raulcd · 2022-04-21T15:42:19Z

@github-actions crossbow submit verify-rc-source-python-linux-conda-latest-amd64

github-actions · 2022-04-21T18:24:13Z

Unable to match any tasks for `conda-verify-rc`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/2202830288```

github-actions · 2022-04-21T18:40:41Z

Revision: 6b96565

Submitted crossbow builds: ursacomputing/crossbow @ actions-1898

Task	Status
verify-rc-source-python-linux-conda-latest-amd64

kou · 2022-04-22T03:36:07Z

Can we add -c safe.directory=... to git command lines in cpp/cmake_modules/DefineOptions.cmake and python/setup.py instead of adding git config --global?

pitrou · 2022-04-22T06:33:40Z

One problem is that AFAICT a git command is implicitly issued by setuptools_scm.
Upstream issue is at pypa/setuptools-scm#707 , but I don't know if they intend to issue a fix.

github-actions · 2022-04-22T06:38:29Z

https://issues.apache.org/jira/browse/ARROW-16219

pitrou · 2022-04-22T07:42:28Z

@kou Apparently one cannot use -c safe.directory: https://git-scm.com/docs/git-config/2.36.0#Documentation/git-config.txt-safedirectory

This config setting is only respected when specified in a system or global config, not when it is specified in a repository config or via the command line option -c safe.directory=.

kou · 2022-04-22T07:47:59Z

@kou Apparently one cannot use -c safe.directory: https://git-scm.com/docs/git-config/2.36.0#Documentation/git-config.txt-safedirectory

This config setting is only respected when specified in a system or global config, not when it is specified in a repository config or via the command line option -c safe.directory=.

Oh, sorry. I missed this.

raulcd · 2022-04-22T08:22:23Z

About the build failures, I am going to fix the Windows ones which are introduced by this PR https://github.com/apache/arrow/runs/6114986154?check_suite_focus=true:

Run export CMAKE_BUILD_PARALLEL_LEVEL=$NUMBER_OF_PROCESSORS
+ source_dir=/d/a/arrow/arrow/cpp
+ build_dir=/d/a/arrow/arrow/build/cpp
+ : ON
+ : OFF
+ git config --global --add safe.directory /d/a/arrow/arrow
ci/scripts/cpp_build.sh: line 28: git: command not found

The MacOS ones are related to homebrew installing git https://github.com/apache/arrow/runs/6114984587?check_suite_focus=true:

 Error: Could not symlink share/doc/git-doc/MyFirstContribution.html
Target /usr/local/share/doc/git-doc/MyFirstContribution.html
is a symlink belonging to [email protected]. You can unlink it:
  brew unlink [email protected]

To force the link and overwrite all conflicting files:
  brew link --overwrite [email protected]

To list all files that would be deleted:
  brew link --overwrite --dry-run [email protected]
Installing git has failed!

raulcd · 2022-04-22T10:47:20Z

@kou this PR fixes all the git failures I've found so far except the brew installation that I've captured on a different ticket (https://issues.apache.org/jira/browse/ARROW-16278) and have started working on a different PR #12956 . Let me know if you think we should do other changes or if I should incorporate the brew fixes here (I am still working on understanding how to fix them).

raulcd · 2022-04-22T15:46:50Z

@kszucs was trying a different approach which seems better, closing this PR in favor of the other one (#12958)

raulcd · 2022-04-22T15:47:31Z

Sorry, this was not the PR to close

kszucs

Thanks @raulcd! Merging on green.

ursabot · 2022-04-26T04:51:07Z

Benchmark runs are scheduled for baseline = ec9440a and contender = c16bbe1. c16bbe1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed] test-mac-arm
[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.55% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/590| c16bbe18 ec2-t3-xlarge-us-east-2>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/578| c16bbe18 test-mac-arm>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/576| c16bbe18 ursa-i9-9960x>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/588| c16bbe18 ursa-thinkcentre-m75q>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/589| ec9440af ec2-t3-xlarge-us-east-2>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/577| ec9440af test-mac-arm>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/575| ec9440af ursa-i9-9960x>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/587| ec9440af ursa-thinkcentre-m75q>
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ARROW-162219: [CI] Fix git config to prevent SCM tools failure findin…

3e9e662

…g the version

ARROW-16219: Fix verify-release-candidate allowing ARROW_SOURCE_DIR a…

26d9eff

…s git safe directory

raulcd marked this pull request as ready for review April 21, 2022 13:48

pitrou reviewed Apr 21, 2022

View reviewed changes

dev/release/verify-release-candidate.sh Outdated Show resolved Hide resolved

ARROW-16219: [CI] Removed git safe directory from verify-release-cand…

6b96565

…idate script and add it to the docker-compose file

pitrou requested a review from kou April 21, 2022 15:54

pitrou changed the title ~~ARROW-162219: [CI] Fix git config to prevent SCM tools failure~~ ARROW-16219: [CI] Fix git config to prevent SCM tools failure Apr 22, 2022

raulcd added 3 commits April 22, 2022 10:32

ARROW-16219: [CI] Ensure git command is present

64d62f5

ARROW-16219: [CI] Fix argument to git command

f843ea7

ARROW-16219: [CI] Attempt to fix ubuntu-lint jobs

5a0d6fe

raulcd mentioned this pull request Apr 22, 2022

ARROW-16278: [CI] Git installation failure on brew #12956

Closed

raulcd closed this Apr 22, 2022

raulcd reopened this Apr 22, 2022

This was referenced Apr 22, 2022

ARROW-15800 [R] Implement bindings for lubridate::as_date() and lubridate::as_datetime() #12738

Closed

ARROW-9235: [R] Support for connection class when reading and writing files #12323

Closed

kszucs approved these changes Apr 22, 2022

View reviewed changes

kszucs closed this in c16bbe1 Apr 22, 2022

ARROW-16219: [CI] Fix git config to prevent SCM tools failure #12945

ARROW-16219: [CI] Fix git config to prevent SCM tools failure #12945

Uh oh!

Conversation

raulcd commented Apr 21, 2022

Uh oh!

raulcd commented Apr 21, 2022

Uh oh!

raulcd commented Apr 21, 2022

Uh oh!

raulcd commented Apr 21, 2022

Uh oh!

Uh oh!

github-actions bot commented Apr 21, 2022

Uh oh!

raulcd commented Apr 21, 2022

Uh oh!

raulcd commented Apr 21, 2022

Uh oh!

github-actions bot commented Apr 21, 2022

Uh oh!

github-actions bot commented Apr 21, 2022

Uh oh!

kou commented Apr 22, 2022

Uh oh!

pitrou commented Apr 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 22, 2022

Uh oh!

pitrou commented Apr 22, 2022

Uh oh!

kou commented Apr 22, 2022

Uh oh!

raulcd commented Apr 22, 2022

Uh oh!

raulcd commented Apr 22, 2022

Uh oh!

raulcd commented Apr 22, 2022

Uh oh!

raulcd commented Apr 22, 2022

Uh oh!

kszucs left a comment

Choose a reason for hiding this comment

Uh oh!

ursabot commented Apr 26, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pitrou commented Apr 22, 2022 •

edited

Loading